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Abstract 


A  question  currently  being  argued  throughout  the  appraisal  community  concerns  evaluation  of 
process  quality  during  Standard  CMMI  Appraisal  Method  for  Process  Improvement 
(SCAMPIsm)  appraisals.  Different,  conflicting  opinions  on  whether  evaluation  of  process  quality 
during  a  SCAMPI  appraisal  is  appropriate  run  from  an  extremely  vigorous  “no”  to  an  equally  vi¬ 
gorous  “yes!” 

The  main  issue  in  both  beliefs  appears  to  be  linked  to  word  definitions — specifically,  a  lack  of 
agreement  among  SCAMPI  Lead  Appraisers  about  what  “quality”  means  in  the  SCAMPI  context. 

This  technical  note  examines  these  beliefs  in  light  of  the  following  questions: 

•  Do  appraisal  results,  following  the  current  methodology,  address  the  quality  of  processes  ex¬ 
amined  during  the  appraisal? 

•  Can  appraisals,  following  the  current  methodology,  judge  the  quality  of  the  final  product  or 
deliverable? 

•  Can  appraisals,  following  the  current  methodology,  provide  insight  into  the  quality  of  the  fi¬ 
nal  product  or  deliverable  during  development? 

Having  addressed  these  questions  in  terms  of  quality,  the  technical  note  discusses  approaches  ac¬ 
quisition  offices  may  use  to  gain  the  insight  desired. 


®  CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
SM  SCAMPI  is  a  service  mark  of  Carnegie  Mellon  University. 
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1  Introduction 


Gaining  insight  into  the  quality  of  acquired  software  products  implies  and  requires  acquisition 
offices  to  (1)  continually  monitor  the  suppliers’  approach  to  “building  in”  the  quality  and  (2)  es¬ 
tablish  a  management  framework  that  supports  the  development  of  a  quality  product.  In  both  cas¬ 
es,  acquisition  offices  must  obtain  such  insight  through  the  request  for  proposal  (RFP)  (or  similar 
acquisition  documents)  and  contracting  processes  with  associated  documentation. 

There  is  currently  a  belief  by  some  acquisition  offices  that  using  process  reference  models  such  as 
the  CMMI  framework  and  applying  associated  appraisals  (such  as  the  Standard  CMMI  Appraisal 
Method  for  Process  Improvement,  or  SCAMPIsm)  will  automatically  yield  the  needed  visibility  or 
insight  to  allow  acquisition  offices  to  manage  the  development  and  therefore  acquire  quality 
products. 

Linked  to  this  belief  are  questions  currently  being  argued  in  the  appraisal  community  concerning 
the  evaluation  of  quality  during  SCAMPI  appraisals.  A  current  philosophy  espoused  by  some  ap¬ 
praisal  teams  appears  to  be  that  the  quality  of  processes  is  not  evaluated.  However,  as  discussed 
below,  this  may  not  be  precisely  correct.  We  believe  the  confusion  can  be  attributed  to  the  unde¬ 
fined  terminology  which  results  in  multiple  understandings  appraisers  have  of  the  word  “quality.” 

This  technical  note  examines  the  various  opinions  in  an  attempt  to  resolve  the  confusion.  We  seek 
an  answer  to  the  following  question: 

Following  the  current  SCAMPI  methodology,  can  and  should  Lead  Appraisers  and  appraisal 
teams  judge  the  quality  of  supplier  processes  examined  during  an  appraisal? 

To  explore  the  question,  we  have  to  understand  several  aspects  surrounding  quality  and  associated 
terminology: 

•  What  do  we  mean  by  quality  with  respect  to  processes  examined  during  an  appraisal? 

•  What  quality  attributes  does  the  SCAMPI  allow  to  be  applied  to  processes,  if  any? 

•  What  would  be  the  effect  of  applying  the  quality  attributes  to  processes  during  a  SCAMPI 
engagement? 

We  postulate  that  the  issues  noted  above  stem  from  the  lack  of  precise  definitions  of  terms  in  ap¬ 
praisal  documentation  related  to  the  word  “quality.”  To  this  end,  we  introduce,  in  Section  2,  terms 
that  establish  a  context  for  the  subsequent  discussions  of  quality.  In  Section  3  we  note  assertions 
made  by  the  appraisal  community  that  reflect  some  differences  in  that  community’s  thinking 
about  what  appraisals  can  accomplish  in  determining  process  quality.  Section  4  examines  these 
opinions  in  light  of  the  current  appraisal  methodology  in  a  SCAMPI  engagement.  Section  5  gives 
a  brief  summary  of  the  discussions  in  the  technical  note. 


®  CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
SM  SCAMPI  is  a  service  mark  of  Carnegie  Mellon  University. 
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2  Terminology 


In  this  section,  we  introduce  and  define  terminology  related  to  the  issue  of  process  evaluation, 
specifically  the  term  quality. 

2.1  Entities — Quality  of  What 

The  first  question  to  answer:  What  entity  are  we  seeking  to  evaluate  for  quality  during  an  apprais¬ 
al?  Baker  identifies  several,  such  as  product,  process,  artifact,  activity,  and  service,  to  which  we 
can  apply  quality  evaluations  [Baker  2007].  Certainly  other  entities  exist  for  this  application;  the 
concept  of  quality  is  generally  applicable  to  almost  anything,  tangible  and  intangible.  Each  entity 
may  have  very  different  quality  attributes  being  evaluated  depending  on  one’s  perspective.  An 
appraiser  must  first  identify  the  entity  or  entities  for  which  the  quality  is  to  be  evaluated.  The  ap¬ 
praiser  must  also  establish  an  evaluation  approach  that  includes  quality  attributes  to  be  used  in  the 
evaluation. 

From  Baker  and  the  question  at  hand,  the  following  is  the  focus  of  our  discussions: 

•  Processes  that  produce  the  artifacts  that  make  up  the  direct  or  indirect  objective  evidence  are 
the  entities  whose  quality  we  wish  to  attempt  to  judge. 

•  The  SCAMPI  Method  Definition  Document  (MDD)  can  provide  the  needed  quality 
attributes  that  can  be  applied  to  the  processes  for  a  more  objective  quality  evaluation  than 
can  be  obtained  through  the  consensus  of  individual  teams. 

•  This  evaluation  approach  can  result  in  support  for  reporting  on  the  quality  of  the  processes  as 
part  of  the  appraisal.  (How  to  use  this  information  is  a  decision  left  to  the  organization  being 
appraised.) 

2.2  Quality  (What  Do  We  Mean  by  Quality?) 

Cooper  provides  an  operational  definition  of  quality: 

Quality  is  the  degree  to  which  an  object  (entity)  [e.g.,  process,  product,  or  service]  satisfies 
a  specified  set  of  attributes  or  requirements  [Cooper  2002]. 

This  identifies  “specified  requirements”  as  the  quality  attributes  to  be  used.  The  attributes  or  re¬ 
quirements  are  to  be  specified  and  must  adhere  to  the  normal  rules  for  requirements  description 
such  as  being  consistent,  clearly  stated,  testable,  or  measurable  (quantitative  in  some  way). 

Significant  here  is  the  concept  of  completely  defining  the  quality  attributes  being  sought.  In  order 
to  understand  the  quality  of  an  entity,  the  context1  for  the  quality  attributes  must  be  defined  and 
specified.  This  concept  is  embedded  in  the  following  statement: 

Just  as  beauty  is  in  the  eye  of  the  beholder,  so  is  quality  [Baker  2007]. 


1  Context  includes  criteria  to  define  quality  in  quantitative  terms  and  the  associated  measurements  of 
quality. 
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Consequently,  a  set  of  attributes  that  one  community  deems  important  as  a  measure  of  quality 
may  not  be  deemed  important  by  another  community.  Rather,  each  community  is  likely  to  have  its 
own  set  of  attributes  and  attribute  values  with  which  to  measure  quality.  This  point  becomes  criti¬ 
cal  when  organizations  are  asked  to  establish  baselines  and  prediction  models  used  to  help  man¬ 
age  statistically  the  product’s  quality  and  process  performance  during  development. 

Overall  we  must  describe  the  entire  context  of  the  quality  we  are  looking  for  to  be  able  to  measure 
or  otherwise  judge  the  quality  of  an  entity. 

We  explore  these  quality  concepts  next  in  the  context  of  appraisals. 
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3  A  Review  of  Appraisers’  Opinions 


To  establish  context  for  our  examination,  we  first  state,  without  analysis,  some  of  the  assertions 
made  by  the  SCAMPI  Lead  Appraiser™  community.  These  assertions  are  examined  later  in  this 
technical  note. 

Some  Lead  Appraisers  assert  the  following: 

While  a  SCAMPI  doesn ’t  measure  effectiveness  or  efficiency  directly,  the  appraisal  team 
does  judge  whether  or  not  the  organization  is  measuring  the  effectiveness  and  efficiency  of 
its  processes.  This  is  done  through  the  generic  practices  (GPs),  product  and  process  quality 
assurance  (PPQA),  organizational  process  focus  (OPF),  organizational  process  perfor¬ 
mance  ( OPP),  and  the  like. 

A  second  position  taken  by  some  Lead  Appraisers  is  as  follows: 

An  appraisal  team  does  indeed  judge  the  “goodness  ”  of  the  process.  The  CMMI  practices 
themselves  are  the  characteristics  of  effective  processes.  The  team  judges  if  these  practices 
are  implemented  using  all  the  informative  material  to  make  that  determination. 

If  a  practice  is  documented  and  not  effective  (or  “good”)  as  executed,  then  the  appraisal 
team  is  failing  to  do  its  job  if  it  doesn ’t  highlight  this  as  a  weakness — either  against  that 
practice,  a  generic  practice,  or  one  of  the  other  areas  listed  earlier. 

Other  Lead  Appraisers  assert  another  position,  reflecting  different  concepts  of  quality  and  quality 
evaluation  during  an  appraisal: 

Artifacts  are  typically  judged  for  “adequacy  ”  as  an  indication  of  their  quality — the  judg¬ 
ments  being  made  by  the  appraisal  team. 

And,  while  the  last  statement  focuses  on  artifacts,  the  philosophy  applies  to  process  quality.  That 
is,  the  evaluation  of  “adequacy”  of  the  process  depends  on  the  experience  of  the  appraisal  team. 
Here  adequacy  is  used  as  one  quality  attribute. 


SM  SCAMPI  Lead  Appraiser  is  a  service  mark  of  Carnegie  Mellon  University. 
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4  Judging  Process  Quality  During  an  Appraisal 


4.1  Process  and  Practices 

Many  of  the  difficulties  in  answering  the  basic  questions  stem  from  the  following  premise  stated 
in  some  of  the  seminal  publications  by  the  SEI  on  process  improvement  [Humphrey  1989]: 

The  quality  of  the  product  is  highly  influenced  by  the  quality  of  the  process  used  to  acquire, 
develop,  and  maintain  the  product. 

This  is  a  fundamental  foundational  premise  for  process  improvement  efforts,  especially  for  Capa- 
bility  Maturity  Model  -based  process  improvement. 

Note  that  the  quality  and  the  context  behind  or  surrounding  the  quality  are  not  defined  for  either 
the  process  or  the  product.  Also  note  that  there  is  no  guarantee  that  a  quality  process,  however  it  is 
defined,  will  produce  a  quality  product;  this  is  reflected  in  the  term  “highly  influenced.  ” 

Based  on  the  definitions,  let’s  review  the  assertions  provided  in  Section  3: 

While  a  SCAMPI  doesn ’t  measure  effectiveness  or  efficiency  directly,  the  appraisal  team 
does  judge  whether  or  not  the  organization  is  measuring  the  effectiveness  and  efficiency  of 
their  processes.  This  is  done  through  GPs,  PPQA,  OPF,  OPP,  and  the  like. 

First,  it  must  be  noted  that  the  general  view  of  the  generic  practices  of  PPQA  is  that  they  only  de¬ 
termine  if  the  process,  as  established,  is  being  followed.  PPQA  does  not  check  for  the  adequacy  of 
the  process.  OPP  may  be  a  better  vehicle,  and  organizational  innovation  and  deployment  (OID)  is 
a  way  of  making  changes  that  quantitatively  show  that  the  process  is  being  improved. 

This  assertion  appears  to  presume  that  the  organization  is  actually  using  measurements  of  the  im¬ 
plemented  practices  from  the  CMMI  or  elsewhere  to  measure  effectiveness  and  efficiency  of  its 
processes.  However,  although  we  would  hope  this  is  the  case,  at  lower  levels  of  maturity,  effi¬ 
ciency  and  effectiveness  may  not  be  defined  by  the  organization  in  any  gainful  way,  let  alone  in  a 
measurable  way.  CMMI-DEV  Generic  Practice  2.8  for  each  process  area  is  intended  to  provide  a 
measure  of  the  process  that  can  be  used  to  monitor  and  control  it  (GP  2.8  is  “monitor  and  control 
the  <xxx>  process  against  the  plan  for  performing  the  process  and  take  appropriate  corrective  ac¬ 
tion”). 

Often,  the  measure  that  is  selected  is  actual  schedule  status  for  that  process  relative  to  the  planned 
schedule.  Such  a  measure  may  not  always  be  adequate  to  monitor  and  control  the  process.  So,  the 
focus  of  the  GPs  and  supporting  process  areas  may  not  yield  desired  results.  (In  general,  we 
would  hope  the  organization  is  measuring  the  efficiency  and  effectiveness,  regardless  of  how  the 
organization  defines  these  attributes,  of  its  own  processes,  to  collect  historical  data  for  creating 
process  performance  baselines  and  models.  This  may  not  be  the  case  unless  the  organization  has 
been  exposed  to  statistical  techniques  and  knows  how  to  apply  them  to  its  processes.)  From  a 
practical  perspective,  most  organizations  are  not  thinking  ahead  in  terms  of  ultimately  achieving 
maturity  level  4  or  5  and  are  more  focused  on  achieving  the  current  lower  target  maturity  level. 


®  Capability  Maturity  Model  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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To  elaborate  further,  it  can  be  said  that  metrics  actually  selected  by  an  organization  may  not  be 
meaningful  or  not  be  particularly  easy  to  define.  The  problem  is  not  what  the  model  requires  but 
how  the  organization  implements  GPs,  specifically  GP  2.8,  as  described  in  the  previous  para¬ 
graph.  Some  organizations  may  choose  something  just  to  satisfy  GP  2.8,  whether  it  is  useful  or 
not,  in  order  to  ensure  they  achieve  largely  implemented"  (LI)  characterization  for  that  practice 
[MDD  2007].  The  problems  being  experienced  with  the  quality  of  the  products  may,  in  some  re¬ 
spects,  be  attributed  to  appraisers’  interpretations  of  the  CMMI. 

A  second  assertion  presents  the  following  argument: 

An  appraisal  team  does  indeed  judge  the  “goodness  ”  of  the  process.  The  CMMI  practices 
themselves  are  the  characteristics  of  effective  processes.  The  team  judges  if  these  practices 
are  implemented  using  all  the  informative  material  to  make  that  determination. 

From  this  statement,  one  may  conclude  that  the  quality  of  an  organization’s  processes  and  practic¬ 
es  could  be  evaluated  by  using  the  CMMI  model  as  a  gauge  of  quality.  That  is,  from  the  above 
statement,  appraisals  can  use  the  CMMI  model  as  a  quality  reference  since  the  model  contains 
“effective”  practices  as  defined  by  the  CMMI  community,  and  adherence  to  these  practices  would 
potentially  be  an  indication  of  process  quality  in  terms  of  effectiveness  and  efficiency  if  these 
attributes  are  properly  defined,  ignoring  the  possibility  that  uncontrollable  outside  factors  may 
also  negate  the  benefits  of  using  a  set  of  best  practices. 

Stated  another  way,  appraisal  teams  use  the  model  to  check  adherence  of  the  organization’s  im¬ 
plemented  processes  and  practices  to  CMMI.  Further,  customers  of  the  appraisal  typically  con¬ 
clude — based  on  the  developer  achieving  a  maturity  or  capability  level  ratings  during  an  apprais¬ 
al — that  since  the  model  is  a  compendium  of  “effective”  processes  or  practices,  adhering  to  them, 
as  shown  in  an  appraisal,  would  be  enough  to  produce  quality  products. 

However,  it  is  not  clear  that  the  adherence  of  the  implemented  practices  to  the  CMMI  practices 
actually  can  be  used  to  “measure”  the  efficiency  and  effectiveness  of  the  related  processes  and 
practices  and  produce  the  desired  quality  product. 

If  a  practice  is  documented  and  not  effective  (or  “good”)  as  executed,  then  the  appraisal 
team  is  failing  to  do  its  job  if  it  doesn ’t  highlight  this  as  a  weakness  either  against  that  prac¬ 
tice,  a  GP,  or  one  of  the  other  areas  listed  earlier. 

It  is  not  clear  from  this  assertion  that  “if  a  practice  is  implemented  by  the  organization”  means  the 
organization’s  process  is  effective  in  producing  what  the  organization  desires  or  needs.  It  is  also 
not  clear  that  the  appraisal  team  can  evaluate  that  a  process  is  effective  for  the  organization  just 
because  the  organization  has  implemented  the  CMMI  practices.  For  example,  there  may  be  many 
more  practices  the  organization  must  implement  within  its  environment  to  make  the  process  effec¬ 
tive  (e.g.,  when  domain  specific  practices,  such  as  those  associated  with  space  versus  ground 
based  systems,  enter  the  equation). 


2  Largely  implemented  is  defined  In  the  MDD  glossary  as  a  practice  characterization  value  assigned  to  a  process 
instantiation  when  (1)  one  or  more  direct  artifacts  are  present  and  judged  to  be  adequate,  (2)  at  least  one  indirect 
artifact  and/or  affirmation  exists  to  confirm  implementation,  and  (3)  one  or  more  weaknesses  are  noted. 
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Much  of  the  confusion  may  be  attributed  to  terminology,  since  terms  like  “efficiency”  and  “effec¬ 
tiveness”  should  be  considered  quality  attributes.  Even  here  these  terms  need  to  be  further  defined 
in  some  way.  And,  the  entire  context  surrounding  these  attributes  needs  to  be  specified. 

Another  view  of  the  issue  is  explored  in  an  article  by  Charette  in  CrossTalk.  (Note  that  Charette 
uses  definitions  different  from  the  CMMI  model  for  process  adherence,  process  capability, 
process  performance,  and  program/project  process  performance.)  He  defines  “. .  .process  adhe¬ 
rence  adequacy  or  performance  as  the  ability  of  an  organization  to  adequately  define  and  imple¬ 
ment  the  technical  and  management  processes  required  of  its  programs”  [Charette  2004]. 

3 

Charette  goes  on  to  describe  two  types  of  process  adherence  shortfalls  found  in  his  assessments 
that  are  different  than  the  maturity  level  achievement  typically  rated  in  a  SCAMPI  appraisal: 

First  are  the  technical  and  management  processes  that  are  poorly  executed — meaning  that 
they  are  ineffectively  implemented  or  performed  for  a  particular  program. 

The  second  type  of process  adherence  shortfall ...  is  constrained  process.  These  are  technic¬ 
al  and  management  processes  that  are  not  fully  implemented  or  executed  because  the  pro¬ 
gram  team  ...no  longer  supports  them 

The  article  never  quite  ties  process  adherence  adequacy  to  these  types  of  process  adherence  short¬ 
falls.  However,  the  reader  is  led  to  make  that  connection. 

More  important  were  his  findings  on  shortfalls  in  process  capabilities  where  the  discussion  on 
judging  the  quality  of  something  during  an  appraisal  comes  into  play. 

Charette  defines  process  capability  as  “. .  ..the  effectiveness  of  the  defined  and  implemented  orga¬ 
nizational  processes  in  meeting  a  specific  program’s  technical  and  management  requirements.  In 
general,  process  capability  refers  to  how  well  an  organization’s  process  models  or  standards  have 
been  adapted  and  applied  to  address  the  specific  characteristics  and  needs  of  a  particular  pro¬ 
gram.” 

This  assertion  resembles  the  CMMI  organizational  maturity  level  4  process  area  of  quantitative 
project  management  (QPM).  QPM  asks  the  project  to  establish  (from  the  organization’s  process 
models  and  baselines  and  from  imposed  requirements  such  as  by  a  contract)  the  quality  and 
process  performance  objectives  for  the  project.  Then,  the  process  area  asks  that  the  project  use 
these  objectives  to  manage  project  execution.  The  definition  in  CMMI  for  process  performance  is 
“a  measure  of  the  actual  results  achieved  by  following  a  process.  It  is  characterized  by  both 
process  measures,  (e.g.,  effort,  cycle  time,  and  defect  removal  efficiency),  and  product  measures 
(e.g.,  reliability,  defect  density,  and  response  time)”  [Chrissis  2006], 

Here,  an  appraisal  does  have  the  ability  to  gain  insight  into  how  the  organization  has  defined 
process  and  product  quality  for  the  project,  where  there  may  be  interim  products,  and  the  final 
product  or  deliverable  from  the  development.  While  the  usual  concern  from  an  acquisition  pers¬ 
pective  is  the  quality  of  the  final  product  or  deliverable,  there  is  always  a  need  by  the  acquisition 
office  to  know  if  there  are  or  will  be  problems  throughout  the  development.  Hence,  the  acquisition 
office  should  be  interested  in  the  process  performance  models  that  the  organization  is  using  to 
provide  a  “heads  up”  on  any  potential  quality  issues  in  the  delivered  product. 

3  Charette  uses  the  term  “assessment.”  His  work  did  not  apply  the  SCAMPI  methodology. 
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Two  significant  points  arise. 

First,  “getting”  to  the  maturity  level  4  aspects  noted  above  is  typically  looked  at  as  conducting  an 
appraisal  to  that  maturity  level.  This  requires  the  organization  and  projects  to  be  at  a  maturity  lev¬ 
el  4  to  “properly”  implement  the  needed  practices.  From  an  acquisition  perspective  it  may  be  seen 
as  only  using  maturity  level  4  organizations — possibly  limiting  competition.  And,  no  quantitative 
link  has  been  established  between  maturity  levels  and  resulting  project  performance  and  product 
quality  [Baker  2007]. 

Secondly,  an  appraisal  reflects  a  point  in  time.  If  an  appraisal  occurs  during  the  development,  it 
typically  will  not  yield  “continuous”  insight  into  the  building  of  quality  into  the  deliverable.  For 
example,  an  appraisal  could  take  place  after  product  coding  phase.  In  this  case,  the  activities  of 
building  quality  into  the  design  (such  as  with  architectural  decisions)  may  be  missed.  On  the  other 
hand,  if  the  appraisal  occurs  before  a  contract  is  signed,  there  will  be  no  insight  into  the  approach 
actually  implemented  and  decisions  made  in  building  quality  into  the  final  deliverable. 

Other  things  being  equal,  appraisals,  even  for  organizational  maturity  level  4,  do  not  provide  the 
desired  insight  for  the  customer  (acquisition  offices)  throughout  the  development.  The  assumption 
here  is  that  there  are  not  numerous  appraisals  for  each  phase  of  development.  However,  appraisal- 
type  methods  could  be  used  to  get  this  insight.  However,  any  of  these  alternative  methods  (such  as 
process  monitoring  or  process  in  execution  reviews),  need  to  be  in  the  RFP  and  contract. 

4.2  Deliverables 

Two  questions  relating  to  the  quality  of  the  final  product  or  deliverables  remain  to  be  addressed. 

1 .  Can  appraisals,  following  the  current  methodology,  judge  the  quality  of  the  final  product 
or  deliverable? 

2.  Can  appraisals,  following  the  current  methodology,  provide  insight  of  the  quality  of  the 
final  product  or  deliverable  during  development? 

On  question  1,  based  on  the  Process  and  Practices  section  above,  appraisals — using  the  current 
methodology — determine  adherence  of  the  processes  to  the  CMMI-DEV  model. 

Combined  with  the  premise  that 

The  quality  of  the  product  is  highly  influenced  by  the  quality  of  the  process  used  to  acquire, 
develop,  and  maintain  the  product 

there  appears  to  be  no  guarantee  that  appraisals  can  determine  that  the  final  product  will  possess 
the  quality  desired  or  specified,  if  the  quality  is  specified  along  with  its  context.  In  fact,  there  is 
ongoing  work  trying  to  make  a  quantitative  connection  [Baker  2007].  Let  us  emphasize  that  this 
premise  applies  to  both  acquisition  and  development  processes  and  there  are  relationships  be¬ 
tween  the  two  sets  of  processes.  What  the  acquirer  does  or  does  not  do  can  impact  the  supplier’s 
implementation  of  the  development  processes. 

Of  course,  determining  quality  after  the  product  is  developed  is  typically  accomplished  through 
verification  and  validation.  But  then  the  quality  is  already  in  the  product. 
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Turning  to  question  2  (can  appraisals,  following  the  current  methodology,  provide  insight  of  the 
quality  of  the  final  product  or  deliverable  during  development?),  there  is  the  potential  for  apprais¬ 
als  to  provide  some  insight  into  the  quality  of  the  final  product  during  development.  This  was  dis¬ 
cussed  in  the  Process  and  Practices  section,  using  the  elements  of  QPM  to  see  if,  through  adhe¬ 
rence  with  the  model,  the  practices  document  the  approach  to  specify  and  manage  the  quality  and 
process  performance. 

At  maturity  level  4  and  using  QPM,  the  development  project  uses  the  organization’s  performance 
models  and  tailors  these  to  set  up  the  project’s  quality  and  process-performance  objectives  which 
“support”  the  needs  and  priorities  of  customers,  end  user,  and  other  stakeholders.  Appraisals,  us¬ 
ing  the  artifacts  from  the  practices,  gain  insight  as  to  the  possibility  that  the  specified  quality  of 
the  final  product  is  going  to  be  achieved.  (In  general,  it  is  thought  that  tailoring  means  “tighten¬ 
ing”  the  baselines  established  by  the  organization.  However,  with  approval  some  projects  may 
“loosen”  the  baseline  as  appropriate  to  the  project  needs,  but  not  more  than  the  customer  require¬ 
ments  allow.) 

Depending  on  the  number  of  appraisals  during  development,  the  use  of  formal  appraisals  for  ma¬ 
turity  level  4  may  be  too  expensive  and  too  late  to  help  make  the  decisions  to  build  the  quality 
into  the  product.  In  case  an  organization  was  appraised  at  maturity  level  4  before  contract  award, 
it  does  not  guarantee  that  the  project  is  following  the  processes  against  which  the  appraisal  was 
accomplished. 

To  overcome  some  of  these  difficulties  with  appraisals,  acquisition  offices  can  evaluate  through 
other  means  during  source  selection  (assuming  a  competitive  situation)  the  developer’s  approach 
to  developing  the  software  product  along  with  specified  quality  and  deciding  if  it  is  reasonable 
and  feasible.  The  acquisition  office  then  must  carry  though  with  trend  analyses  of  the  developer’s 
various  appropriate  processes  [Cooper  2002]. 

For  maturity  level  4  development  organizations,  the  acquisition  office  must  also  have  the  capabili¬ 
ty  to  understand  statistical  techniques  that  the  contractors  may  be  using  for  managing  the  software 
development. 

In  general  for  this  approach,  acquisition  offices  must  do  the  following: 

•  Specify  the  desired  quality  attributes  of  the  final  product,  or  specify  in  the  RFP  or  statement 
of  work  that  the  developer  will  determine  quantitatively  the  quality  attributes  as  part  of  the 
development  approach,  with  acquisition  office  approvals. 

•  Add  the  correct  wording  to  the  RFP  to  allow  evaluation  of  the  development  approach  during 
source  selection  and  then  allowing  the  acquisition  offices  access  to  results  of  the  processes 
followed,  so  that  trend  analysis  can  be  implemented.  In  this  way,  it  may  be  possible  to  call 
out  the  CMMI  level  4  process  area  of  QPM  and  call  out  the  CMMI  level  3  process  area  of 
technical  solution  to  help  ensure  the  requisite  quality  is  built  in. 

•  Add  words  to  the  RFP  and  to  the  resulting  contract  as  appropriate  to  be  able  to  evaluate  the 
developer’s  approach  to  overall  software  development.  Wording  may  be  in  the  form  of  a 
software  development  plan  (SDP).  Such  an  approach  to  source  selection  and  contract  man¬ 
agement  has  been  suggested  in  the  past  in  the  SA-CMM  VI. 03  and  the  Navy  memorandum 
[Cooper  2002,  Navy  2006].  In  source  selection,  appropriate  wording  for  Evaluation  Factors 
for  Award  is  required  to  ensure  discrimination  among  offers. 
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Add  words  to  the  RFP  and  to  the  resulting  contract  as  appropriate  to  allow  initial  evaluation 
of  the  developer’s  approach  to  building  quality  into  the  product  and  allow  the  acquisition  of¬ 
fice  access  to  appropriate  documentation.  Request  that  the  potential  development  organiza¬ 
tions  describe  in  detail  the  plans,  tools,  methods,  practices,  processes,  and  prediction  models 
that  will  provide  the  insight  into  deliverable  product  quality  throughout  the  development 
project  life  cycle — not  simply  at  the  end  of  the  project.  Tests  can  be  made  at  the  end  of  the 
project,  but  that  is  typically  too  late.  In  source  selection,  appropriate  wording  for  Evaluation 
Factors  for  Award  is  required  to  ensure  discrimination  among  offerors. 
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5  Summary 


Appraisals  do  not  necessarily  give  a  good  indication  of  the  quality  of  the  processes  used  for  prod¬ 
uct  development,  only  adherence  to  the  reference  model.  In  this  technical  note  we  concentrated  on 
the  CMMI-DEV  as  a  reference  model  and  the  SCAMPI  appraisal  method. 

Appraisals  can  provide  some  insight  into  deliverable  product  quality.  However,  the  insight  can  be 
somewhat  indirect.  That  is,  appraisals  compare  the  organization’s  implementation  of  the  CMMI- 
DEV  practices  and  the  adherence  of  the  implementation  of  these  practices  to  the  reference  model. 

This  comparison  evaluates  the  process  adherence  and  not  necessarily  the  quality  of  the 
processes  and  resultant  product.  (A  tacit  assumption  is  that  the  development  organization  has 
processes  that  not  only  adhere  to  the  practices  in  CMMI-DEV  but  also  contribute  to  the  organ¬ 
ization  ’s  business  objectives .) 

However,  the  process  premise  itself,  upon  which  many  process  improvement  programs  are  based, 
implies  that  the  quality  of  a  process  does  not  guarantee  the  needed  product  quality  will  be 
achieved.  (Here,  we  have  not  defined  “needed.”)  Another  way  of  saying  this  is  that  appraisals  do 
not  guarantee  that  the  quality  of  the  deliverable  product  is  built  in  by  the  development  processes 
used.  Thus,  using  only  an  appraisal  does  not  necessarily  give  you  insight  into  product  quality, 
especially  throughout  the  development. 

On  the  other  hand,  we  discussed  how  one  might  get  this  insight  by  using  CMMI-DEV  organiza¬ 
tion  maturity  level  4  process  areas  of  OPP  and  QPM.  While  an  appraisal,  even  at  organizational 
maturity  level  4,  can  be  used  to  obtain  more  insight  into  deliverable  product  quality,  one  cannot 
assume  that  it  guarantees  you  the  insight.  You  must  continue  to  analyze  the  developer’s  process 
and  practices  from  the  start  and  throughout  the  development.  That  continual  analysis  is  fundamen¬ 
tal  to  QPM.4 

During  an  appraisal,  one  can  use  the  organization’s  or  project’s  own  quality  and  performance  ob¬ 
jectives  to  assess  or  judge  the  adequacy  of  the  implemented  process  and  practices  to  produce  a 
quality  work  product,  rather  than  relying  on  the  appraisal  team  experience  and  common  sense  to 
evaluate  the  quality  of  the  work  products  or  final  products.  However,  the  objectives  have  to  be 
established  for  anything  to  be  judged.  And  the  objectives  may  not  be  established  for  all  processes. 

The  recommendation  is  for  the  acquisition  office  to  generate  request  for  proposal,  statement  of 
work,  and  contract  language  to  ensure  the  needed  insight  is  obtained  throughout  development. 
Critically,  the  acquisition  offices  must  follow  through  during  development  to  gain  and  exploit  this 
insight  into  processes  and  resultant  work  product  quality  throughout  the  development  to  help 
make  the  decisions  that  affect  the  quality  of  the  final  product. 


There  is  a  potential  problem  here  in  relying  on  a  maturity  level  4  appraisal.  It  assumes  that  the  development 
organization  is  at  maturity  level  4,  and  this  may  limit  what  development  organization  can  bid  on  the  acquisition 
(i.e.,  limit  competition). 
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